The terminology “subpixel” or “subpixels” as used herein refers to one or more components of a single pixel. In a color subpixelated image, the components of a single pixel comprise several primary color primaries, which may be three ordered color elements such as blue, green, and red (BGR), or red, green, and blue (RGB), or green, blue, red. Some image displays have more than three primaries, further including yellow (RGBY), or white (RGBW), or yellow and cyan (RGBYC). Sub-pixels may appear as a single color when viewed directly by the human eye due to spatial integration by the eye. However, the components may be visible under a magnification. Upon achieving a predetermined resolution, the colors in the sub-pixels are not visible, but the relative intensity of the components may cause a shift in the apparent position or orientation of a line. The resolution at which sub-pixels go unnoticed may depend upon the viewer or the application as the visual processes of humans vary; whereby the colored “fringes” resulting from sub-pixel rendering may cause distraction in some individuals or applications but not others. Methods that account for sub-pixel rendering are generally called subpixel rendering algorithms.
Image resolution relates to the detail that an image possesses. For satellite images, the resolution generally correlates to the area represented by each pixel. Generally speaking, an image is considered to be more accurate and detailed as the area represented by each pixel is decreased. As used herein, the term images includes digital images, film images, and/or other types of images.
As used herein, the terminology “sampling” refers to the practice of collecting a subset of individual observations intended to yield some knowledge about a subject. “Sampling” includes the act, process, or technique of selecting a representative part of a population or sequence of moving images for the purpose of determining parameters or characteristics of the whole population or moving image sequence. For example, according to the 1949 sampling theorem by C E Shannon, in order to recover a signal function ƒ(t) precisely, the sampling rate must be done at a rate greater than twice the signal's highest frequency component.
Many low-cost sensors (or cameras) may spatially or electronically undersample an image. Similarly, cameras taking pictures from great distances, such as aerial photos, may not obtain detailed information about the subject matter. This may result in aliased images in which the high frequency components are folded into the low frequency components in the image. Consequently, subtle or detail information (high frequency components) are not present in the images.
Many digital cameras acquire images using a single image sensor overlaid with a color filter array (CFA), and demosaicing is required to render these images into a viewable format. “Demosaicing process” is a digital image process of reconstructing a full color image from incomplete color samples output from an image sensor or camera overlaid with a color filter array (CFA).
When an image is captured by a monochrome camera, a single charge-coupled device (CCD) or complementary metal-oxide semiconductor (CMOS) sensor is used to sample the light intensity projected onto the sensor. Color images are captured in much the same way, except that the light intensity is measured in separate color channels, usually red, green, and blue. In order to do this, three separate sensors could be used in conjunction with a beam splitter to accurately measure each of the three primary colors at each pixel. However, this approach is expensive and mechanically difficult to implement, making its use in commercial imaging systems infeasible. To overcome this obstacle, the color filter array (CFA) was introduced to capture a color image using only one sensor.
A CFA is an array that is used in front of the image sensors to alternate color filters that samples only one color channel at each pixel location. The most popular and common CFA is arranged in mosaic pattern, called the Bayer pattern, as described in U.S. Pat. No. 3,971,065, “Color image array,” July 1976, B. E. Bayer, which is illustrated in FIG. 1. For each 2×2 set of pixels, two diagonally opposed pixels have green filters, and the other two pixels have red and blue filters, which is called quincunx sampling pattern. A Bayer image can be also seen as a grayscale image as shown in FIG. 28.
This pattern results in half of the image resolution being dedicated to accurate measurement of the green color channel and quarter of the image resolution of the red or blue color channel. The peak sensitivity of the human visual system lies in the medium wavelengths, justifying the extra green sampling as described in X. Li, B. Gunturk, and L. Zhang, “Image demosaicing: a systematic survey,” Proc. SPIE, Vol. 6822, 68221J, 2008. Because each pixel now has only one color sampled, in order to produce a three-channel full color image, that is, each pixel contains three color channel values, missing information needs to be estimated from surrounding pixels of CFA pattern raw data. This is called demosaicing algorithm or process. The simplest demosaicing algorithm is linear interpolation applied to every color channel. More advanced demosaicing methods are summarized in X. Li, B. Gunturk, and L. Zhang, “Image demosaicing: a systematic survey,” Proc. SPIE, Vol. 6822, 68221J, 2008; D. Alleysson and B. C. de Lavarene, “Frequency selection demosaicking: a review and a look ahead,” Proc. SPIE-IS&T Electronic Imaging, Vol. 6822, 68221M, 2008.
Sampling by color filter array (CFA) causes severe aliasing in addition to the aliasing caused by undersampling of many low-cost sensors, e.g., the CCD aperture.
The resolution of a color image reconstructed from the demosaicing method is equal to the physical resolution of a CCD. However, the resolution of an image sensor can be improved by a digital image processing algorithm: super-resolution image reconstruction.
Super-resolution image reconstruction generally increases image resolution without necessitating a change in the design of the optics and/or detectors by using a sequence (or a few snapshots) of low-resolution images. Super-resolution image reconstruction algorithms effectively de-alias undersampled images to obtain a substantially alias-free or, as identified in the literature, a super-resolved image.
When undersampled images have sub-pixel shifts between successive frames, they contain different information regarding the same scene. Super-resolution image reconstruction involves, inter alia, combining information contained in undersampled images to obtain an alias-free (high-resolution) image. Super-resolution image reconstruction from multiple snapshots, taken by a detector which has shifted in position, provides far more detail information than any interpolated image from a single snapshot.
Methods for super-resolving images that are acquired from CFA sensors are in the following references.
One of these methods is to capture multiple appropriately positioned CFA images that are used to fill in the “holes” in a Bayer pattern, as for example in U.S. Pat. No. 7,218,751, “Generating super resolution digital images,” May 15, 2007, A. M. Reed and B. T. Hannigan. Values of the pixels in multiple images which are appropriately aligned to each pixel position are averaged to generate a better value for each pixel position. In this method, in order to produce a useful result, information carried by a digital watermark is used to determine the alignment of the images.
Another type of these methods is called separate approach or two-pass algorithm. The CFA images are first demosaicked and then followed by the application of super-resolution, as for example in U.S. Pat. No. 7,260,277, “Method for obtaining a high-resolution digital image,” Aug. 21, 2007, G. Messina, S. Battiato, and M. Mancuso. Each CFA input image is subjected to an interpolation phase to generate a complete low-resolution with three color channels containing three primary colors in RGB format and is thus linearly transformed into a complete low-resolution image in the YCrCb format, where Y represents the luminance component, Cr and Cb represent two chrominance components. After this, a modified back projection super-resolution approach, based in M. Irani and S. Peleg, “Super resolution from image sequence,” Proceedings of the 10th International Conference on Pattern Recognition, Vol. 2, pages 115-120, is applied only to the luminance component Y of the multiple images.
The third type of these methods is called joint or one-pass method in which demosaicing and super-resolution algorithms are simultaneously carried for a CFA image sequence, as for example in U.S. Pat. No. 7,515,747, “Method for creating high resolution color image, system for creating high resolution color image and program creating high resolution color image,” Apr. 7, 2009, M. Okutomi and T. Goto; U.S. Pat. No. 7,379,612, “Dynamic reconstruction of high-resolution video from color-filtered low-resolution video-to-video super-resolution,” May 27, 2008, P. Milanfar, S. Farsiu, and M. Elad. In this method, an observation model is formulated to relate the original high-resolution image to the observed low-resolution images to incorporate color measurements encountered in video sequences. Then, the high-resolution full color image is obtained by solving an inverse problem by minimizing a cost function which is the function of the difference between the estimated low-resolution input images and the measured input images by imposing penalty terms, such as smoothness of chrominance and inter-color dependencies.
There is a need to produce high-resolution three channel full color images from a sequence of low-resolution images captured by a low-cost CFA imaging device that has experienced translations and/or rotations. The amount of sub-pixel translation and/or rotation may be unknown and it creates a need to estimate sub-pixel translation and rotation. In order to produce high-resolution images from a sequence of CFA undersampled (low-resolution) images, there exists a need to eliminate aliasing, while taking into account natural jitter.